{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tomasonjo/blogs/blob/master/pubmed/Pubmed%20NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "20e0d7eb",
      "metadata": {
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "id": "20e0d7eb",
        "outputId": "236c8be7-00ed-40fb-c408-75b5b73ac9da"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-b1166e23-9796-40ff-bfe8-e88c201dfd38\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-b1166e23-9796-40ff-bfe8-e88c201dfd38\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving spark_nlp_for_healthcare_spark_ocr_4435.json to spark_nlp_for_healthcare_spark_ocr_4435.json\n"
          ]
        }
      ],
      "source": [
        "# Import John Snow License keys\n",
        "import json\n",
        "\n",
        "from google.colab import files\n",
        "\n",
        "license_keys = files.upload()\n",
        "\n",
        "with open(list(license_keys.keys())[0]) as f:\n",
        "    license_keys = json.load(f)\n",
        "\n",
        "# Defining license key-value pairs as local variables\n",
        "locals().update(license_keys)\n",
        "\n",
        "# Adding license key-value pairs to environment variables\n",
        "import os\n",
        "os.environ.update(license_keys)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "34c5bef9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "34c5bef9",
        "outputId": "7c69d490-2378-4bd9-a9f1-d0965e57870f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K     |████████████████████████████████| 140 kB 3.2 MB/s \n",
            "\u001b[K     |████████████████████████████████| 198 kB 7.2 MB/s \n",
            "\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[K     |████████████████████████████████| 147 kB 404 kB/s \n",
            "\u001b[?25hRequirement already satisfied: neo4j in /usr/local/lib/python3.7/dist-packages (4.4.1)\n",
            "Requirement already satisfied: xmltodict in /usr/local/lib/python3.7/dist-packages (0.12.0)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from neo4j) (2018.9)\n"
          ]
        }
      ],
      "source": [
        "# Installing pyspark and spark-nlp\n",
        "! pip install --upgrade -q pyspark==3.1.2 spark-nlp==$PUBLIC_VERSION\n",
        "\n",
        "# Installing Spark NLP Healthcare\n",
        "! pip install --upgrade -q spark-nlp-jsl==$JSL_VERSION  --extra-index-url https://pypi.johnsnowlabs.com/$SECRET\n",
        "\n",
        "# Installing Spark NLP Display Library for visualization\n",
        "! pip install -q spark-nlp-display\n",
        "# Installing neo4j driver and xml parser\n",
        "! pip install neo4j xmltodict"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Agenda\n",
        "In this post, we will start by downloading biomedical articles from PubMed. The PubMed provides an API to retrieve data as well as a FTP site, where daily updates are available.\n",
        "Next, we will run the data through an NLP pipeline to extract relationships between biomedical entities. There are many open-source named entity recognition models out there, but unfortunately, I haven't come across any biomedical relation extraction models that don't require manual training. Since the goal of this post is not to teach you how to train a biomedical relation extraction model but rather how to apply it to solve real-world problems, we will be using the John Snow Labs Healthcare models. [John Snow Labs](https://www.johnsnowlabs.com/) offer free models for recognizing entities and extracting relations from news-like text. However, the biomedical models are not open-source. Luckily for us, they offer a free 30-day trial period for healthcare models. To follow along with the examples in this post, you will need to start the free trial and obtain the license keys.\n",
        "In the last part of this post, we will store the extracted relations in Neo4j, a native graph database designed to store and analyze highly interconnected data. I will also explain some considerations regarding the different graph models we can use to represent the data.\n",
        "# Steps\n",
        "* Download and parse daily update of articles from the PubMed FTP site\n",
        "* Store articles in Neo4j\n",
        "* Use John Snow Labs models to extract relations from text\n",
        "* Store and analyze relations in Neo4j\n",
        "\n",
        "# Download daily update from the PubMed FTP site\n",
        "As mentioned, the PubMed daily updates are available on their FTP site. The data is available in XML format. The files have an incremental ID. I've first tried to calculate the incremental file id for a specific date programmatically. However, it's not straightforward, and I didn't want to waste my time figuring it out, so you will have to copy the desired file location in the code manually."
      ],
      "metadata": {
        "id": "lelg5ZptTDhV"
      },
      "id": "lelg5ZptTDhV"
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "167e3b39",
      "metadata": {
        "id": "167e3b39"
      },
      "outputs": [],
      "source": [
        "import urllib\n",
        "import gzip\n",
        "import io\n",
        "import xmltodict\n",
        "\n",
        "# Get latest pubmed daily update location at\n",
        "# https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/\n",
        "\n",
        "url = \"https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/pubmed22n1211.xml.gz\"\n",
        "oec = xmltodict.parse(gzip.GzipFile(fileobj=io.BytesIO(urllib.request.urlopen(url).read())))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My gut instinct was that it would be easier to convert the XML to a dictionary and process that. However, if I had to do it again, I would probably use XML search functions as I had to include several exceptions to extract required data from the dictionary format correctly."
      ],
      "metadata": {
        "id": "HV9RhMWgT6d0"
      },
      "id": "HV9RhMWgT6d0"
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "741886f1",
      "metadata": {
        "id": "741886f1"
      },
      "outputs": [],
      "source": [
        "from datetime import date\n",
        "\n",
        "# Export pubmed article params\n",
        "params = list()\n",
        "\n",
        "for row in oec['PubmedArticleSet']['PubmedArticle']:\n",
        "\n",
        "    # Skip articles without abstract or other text\n",
        "    if not row['MedlineCitation']['Article'].get('Abstract'):\n",
        "        continue\n",
        "\n",
        "    # Article id\n",
        "    pmid = row['MedlineCitation']['PMID']['#text']\n",
        "\n",
        "    abstract_raw = row['MedlineCitation']['Article']['Abstract']['AbstractText']\n",
        "\n",
        "    if isinstance(abstract_raw, str):\n",
        "        text = [{'label': 'SINGLE', 'text': abstract_raw}]\n",
        "    elif isinstance(abstract_raw, list):\n",
        "        text = [{'label': el.get('@Label', 'SINGLE'), 'text': el['#text']}\n",
        "                for el in abstract_raw if not isinstance(el, str) and el.get('#text')]\n",
        "    else:\n",
        "        text = [{'label': abstract_raw.get(\n",
        "            '@Label', 'SINGLE'), 'text': abstract_raw.get('#text')}]\n",
        "\n",
        "    # Completed date\n",
        "    if row['MedlineCitation'].get('DateCompleted'):\n",
        "        completed_year = int(row['MedlineCitation']['DateCompleted']['Year'])\n",
        "        completed_month = int(row['MedlineCitation']['DateCompleted']['Month'])\n",
        "        completed_day = int(row['MedlineCitation']['DateCompleted']['Day'])\n",
        "        completed_date = date(completed_year, completed_month, completed_day)\n",
        "    else:\n",
        "        completed_date = None\n",
        "\n",
        "    # Revised date\n",
        "    revised_year = int(row['MedlineCitation']['DateRevised']['Year'])\n",
        "    revised_month = int(row['MedlineCitation']['DateRevised']['Month'])\n",
        "    revised_day = int(row['MedlineCitation']['DateRevised']['Day'])\n",
        "    revised_date = date(revised_year, revised_month, revised_day)\n",
        "\n",
        "    # title\n",
        "    title_raw = row['MedlineCitation']['Article']['ArticleTitle']\n",
        "    if isinstance(title_raw, str):\n",
        "        title = title_raw\n",
        "    else:\n",
        "        title = title_raw['#text'] if title_raw else None\n",
        "    # Country\n",
        "    country = row['MedlineCitation']['MedlineJournalInfo']['Country']\n",
        "\n",
        "    # Mesh headings\n",
        "    mesh_raw = row['MedlineCitation'].get('MeshHeadingList')\n",
        "    if mesh_raw:\n",
        "        if isinstance(mesh_raw['MeshHeading'], list):\n",
        "            mesh = [{'mesh_id': el['DescriptorName']['@UI'], 'text': el['DescriptorName']['#text'], 'major_topic': el['DescriptorName']\n",
        "                     ['@MajorTopicYN']} for el in mesh_raw['MeshHeading']]\n",
        "        else:\n",
        "            mesh = [{'mesh_id': el['DescriptorName']['@UI'], 'text': el['DescriptorName']['#text'], 'major_topic': el['DescriptorName']\n",
        "                     ['@MajorTopicYN']} for el in [mesh_raw['MeshHeading']]]\n",
        "    else:\n",
        "        mesh = []\n",
        "\n",
        "    # Authors\n",
        "    authors_raw = row['MedlineCitation']['Article'].get('AuthorList')\n",
        "    if not authors_raw:\n",
        "        authors = []\n",
        "    elif isinstance(authors_raw['Author'], list):\n",
        "        authors = [\n",
        "            f\"{el['ForeName']} {el['LastName']}\" for el in authors_raw['Author'] if el.get('ForeName')]\n",
        "    else:\n",
        "        authors = [f\"{authors_raw['Author']['ForeName']} {authors_raw['Author']['LastName']}\"] if authors_raw['Author'].get('ForeName') else None\n",
        "\n",
        "    params.append({'pmid': pmid, 'text': text, 'completed_date': completed_date,\n",
        "                  'revised_date': revised_date, 'title': title, 'country': country, 'mesh': mesh, 'author': authors})\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Store articles in Neo4j\n",
        "Before moving onto the NLP extraction pipeline, we will store the articles in Neo4j.\n",
        "In the center of the graph are the articles. We store their PubMed ids, title, country, and dates as properties. Of course, we could refactor the country as a separate node if we wanted to, but here I modeled them as node properties. Each article contains one or more sections of texts. Several types of sections are available, like the abstract, methods, or conclusions. I've stored the section type as the relationship property between the article and the section. We also know who authored a particular research paper. PubMed articles in particular also contain the entities mentioned or researched in the paper, which we will store as the Mesh node as the entities are mapped to the Mesh ontology.\n",
        "P.s. For most articles, only the abstract is available. You could probably download full-text for most articles through the PubMed API. However, we won't do that here.\n",
        "Before importing the data, we have to set up our Neo4j environment. If you are using the Colab notebook, I suggest you open a [Blank Project in Neo4j Sandbox](https://sandbox.neo4j.com/?usecase=blank-sandbox). Neo4j Sandbox is a free time-limited cloud instance of Neo4j. Otherwise, if you want a local Neo4j environment, I suggest you download and install the [Neo4j Desktop](https://neo4j.com/download/) application. Make sure you install the APOC library in your local environment. \n",
        "Once you have set up your Neo4j instance, copy the connection details to script."
      ],
      "metadata": {
        "id": "G8J6FkBOT9wU"
      },
      "id": "G8J6FkBOT9wU"
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "923451ad",
      "metadata": {
        "id": "923451ad"
      },
      "outputs": [],
      "source": [
        "# Define Neo4j connections\n",
        "import pandas as pd\n",
        "from neo4j import GraphDatabase\n",
        "host = 'bolt://54.89.97.91:7687'\n",
        "user = 'neo4j'\n",
        "password = 'witnesses-bells-drunk'\n",
        "driver = GraphDatabase.driver(host,auth=(user, password))\n",
        "\n",
        "def run_query(query, params={}):\n",
        "    with driver.session() as session:\n",
        "        result = session.run(query, params)\n",
        "        return pd.DataFrame([r.values() for r in result], columns=result.keys())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A good practice when dealing with Neo4j is to define unique constraints and indexes to optimize the performance of both import and read queries."
      ],
      "metadata": {
        "id": "_G0-IyWRUX_Y"
      },
      "id": "_G0-IyWRUX_Y"
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "69777f31",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49
        },
        "id": "69777f31",
        "outputId": "8458f27f-3946-4ee4-81e2-e352fe58aee9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-f70e262a-4a87-4c16-8802-3a6c8ec1b05d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f70e262a-4a87-4c16-8802-3a6c8ec1b05d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f70e262a-4a87-4c16-8802-3a6c8ec1b05d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f70e262a-4a87-4c16-8802-3a6c8ec1b05d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "# Define constraints\n",
        "run_query(\"CREATE CONSTRAINT IF NOT EXISTS ON (a:Article) ASSERT a.pmid IS UNIQUE;\")\n",
        "run_query(\"CREATE CONSTRAINT IF NOT EXISTS ON (a:Author) ASSERT a.name IS UNIQUE;\")\n",
        "run_query(\"CREATE CONSTRAINT IF NOT EXISTS ON (m:Mesh) ASSERT m.id IS UNIQUE;\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we are all set, we can go ahead and import articles into Neo4j."
      ],
      "metadata": {
        "id": "Piby7m5eUaIv"
      },
      "id": "Piby7m5eUaIv"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "c598874a",
      "metadata": {
        "id": "c598874a"
      },
      "outputs": [],
      "source": [
        "import_pubmed_query = \"\"\"\n",
        "UNWIND $data AS row\n",
        "// Store article\n",
        "MERGE (a:Article {pmid: row.pmid})\n",
        "SET a.completed_date = date(row.completed_date),\n",
        "    a.revised_date = date(row.revised_date),\n",
        "    a.title = row.title,\n",
        "    a.country = row.country\n",
        "// Store sections of articles\n",
        "FOREACH (map IN row.text | \n",
        "    CREATE (a)-[r:HAS_SECTION]->(text:Section)\n",
        "    SET text.text = map.text,\n",
        "        r.type = map.label)\n",
        "// Store Mesh headings        \n",
        "FOREACH (heading IN row.mesh | \n",
        "    MERGE (m:Mesh {id: heading.mesh_id})\n",
        "    ON CREATE SET m.text = heading.text\n",
        "    MERGE (a)-[r:MENTIONS_MESH]->(m)\n",
        "    SET r.isMajor = heading.major_topic)\n",
        "// Store authors    \n",
        "FOREACH (author IN row.author | \n",
        "    MERGE (au:Author {name: author})\n",
        "    MERGE (a)<-[:AUTHORED]-(au))\n",
        "\"\"\"\n",
        "\n",
        "# Import pubmed articles into Neo4j\n",
        "step = 1000\n",
        "for x in range(0, len(params), step):\n",
        "    chunk = params[x:x+step]\n",
        "    try:\n",
        "        run_query(import_pubmed_query, {'data': chunk})\n",
        "    except Exception as e:\n",
        "        print(e)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The import is split into batches of 1000 articles to avoid dealing with a single huge transaction and potential memory issues. The import Cypher statement is a bit longer, but nothing too complex. We can quickly inspect the data before moving on to the NLP pipeline."
      ],
      "metadata": {
        "id": "JI52zCkCUdvv"
      },
      "id": "JI52zCkCUdvv"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (a:Article)\n",
        "RETURN count(*) AS count\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "3b9fp76kouzG",
        "outputId": "48d2e47e-e15e-4c95-9836-69ff2138d2cb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        }
      },
      "id": "3b9fp76kouzG",
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-8140eb72-ee71-473e-aa75-fce6efe1918b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>26829</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8140eb72-ee71-473e-aa75-fce6efe1918b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8140eb72-ee71-473e-aa75-fce6efe1918b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8140eb72-ee71-473e-aa75-fce6efe1918b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   count\n",
              "0  26829"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can compare the revised versus the completed date to understand better why there are so many articles."
      ],
      "metadata": {
        "id": "77kNDFc7UiQj"
      },
      "id": "77kNDFc7UiQj"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (a:Article)\n",
        "RETURN a.pmid AS article_id, a.completed_date AS completed_date, a.revised_date AS revised_date\n",
        "ORDER BY completed_date ASC\n",
        "LIMIT 5\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "mFZWa-11o-6v",
        "outputId": "d2271be7-6e1d-4d53-952f-f7b3eb936b5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "id": "mFZWa-11o-6v",
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-f9606d21-47b5-4d18-8ee1-63cf4c8cd486\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>article_id</th>\n",
              "      <th>completed_date</th>\n",
              "      <th>revised_date</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>10954585</td>\n",
              "      <td>2000-10-30</td>\n",
              "      <td>2022-02-28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>11802252</td>\n",
              "      <td>2002-03-15</td>\n",
              "      <td>2022-02-28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>18254086</td>\n",
              "      <td>2008-04-14</td>\n",
              "      <td>2022-02-28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>18646090</td>\n",
              "      <td>2008-10-15</td>\n",
              "      <td>2022-02-28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>19093323</td>\n",
              "      <td>2009-02-03</td>\n",
              "      <td>2022-02-28</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f9606d21-47b5-4d18-8ee1-63cf4c8cd486')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f9606d21-47b5-4d18-8ee1-63cf4c8cd486 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f9606d21-47b5-4d18-8ee1-63cf4c8cd486');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "  article_id completed_date revised_date\n",
              "0   10954585     2000-10-30   2022-02-28\n",
              "1   11802252     2002-03-15   2022-02-28\n",
              "2   18254086     2008-04-14   2022-02-28\n",
              "3   18646090     2008-10-15   2022-02-28\n",
              "4   19093323     2009-02-03   2022-02-28"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "I have no idea why articles older than 20 years are being revised, but we get that information from the XML files. Next, we can inspect which mesh entities are most frequently researched as major topics in the articles completed in 2020 or later."
      ],
      "metadata": {
        "id": "TGI-KXHIUkzD"
      },
      "id": "TGI-KXHIUkzD"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (a:Article)-[rel:MENTIONS_MESH]->(mesh_entity)\n",
        "WHERE a.completed_date.year >= 2020 AND rel.isMajor = \"Y\"\n",
        "RETURN mesh_entity.text as entity, count(*) AS count\n",
        "ORDER BY count DESC\n",
        "LIMIT 5\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "fFOaxQBapSS-",
        "outputId": "c6ff8e0e-5d0d-42bb-90f6-8eb522f24958",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "id": "fFOaxQBapSS-",
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-1343c604-b312-4b51-bf4a-e63d4934196b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>entity</th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>33</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>HIV Infections</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Alcoholism</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Influenza Vaccines</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Diabetes Mellitus, Type 2</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1343c604-b312-4b51-bf4a-e63d4934196b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1343c604-b312-4b51-bf4a-e63d4934196b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1343c604-b312-4b51-bf4a-e63d4934196b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                      entity  count\n",
              "0                   COVID-19     33\n",
              "1             HIV Infections     10\n",
              "2                 Alcoholism      8\n",
              "3         Influenza Vaccines      7\n",
              "4  Diabetes Mellitus, Type 2      7"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Interestingly, COVID-19 comes out on top even though we imported only a single daily update. Before relation extraction NLP models gained popularity, you could use co-occurrence networks to identify potential links between entities. For example, we can inspect which entities most frequently co-occur with COVID-19."
      ],
      "metadata": {
        "id": "5bM_Uzp_Um4H"
      },
      "id": "5bM_Uzp_Um4H"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (e1:Mesh)<-[:MENTIONS_MESH]-(a:Article)-[:MENTIONS_MESH]->(e2)\n",
        "WHERE e1.text = 'COVID-19'\n",
        "RETURN e1.text AS entity1, e2.text AS entity2, count(*) AS count\n",
        "ORDER BY count DESC\n",
        "LIMIT 5\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "kfqmXZkKq0oG",
        "outputId": "640902d0-98db-4875-da7f-984e0ba77d48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "id": "kfqmXZkKq0oG",
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-d9db16a8-75e0-4022-80f8-879551749b73\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>entity1</th>\n",
              "      <th>entity2</th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>Humans</td>\n",
              "      <td>58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>SARS-CoV-2</td>\n",
              "      <td>50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>Female</td>\n",
              "      <td>24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>Male</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>COVID-19</td>\n",
              "      <td>Pandemics</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d9db16a8-75e0-4022-80f8-879551749b73')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d9db16a8-75e0-4022-80f8-879551749b73 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d9db16a8-75e0-4022-80f8-879551749b73');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "    entity1     entity2  count\n",
              "0  COVID-19      Humans     58\n",
              "1  COVID-19  SARS-CoV-2     50\n",
              "2  COVID-19      Female     24\n",
              "3  COVID-19        Male     22\n",
              "4  COVID-19   Pandemics     20"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Co-occurrence results for COVID-19 make sense, even though they don't explain much other than it's related to humans and pandemics and has a strong connection to SARS-CoV-2.\n",
        "# Relation Extraction NLP pipeline\n",
        "Simple co-occurrence analysis can be a powerful technique to analyse relations between entities, but it ignores a lot of information that is available in the text. For that reason, researches have been investing a lot of effort in building in training relation extraction models."
      ],
      "metadata": {
        "id": "um2dWIsmUqff"
      },
      "id": "um2dWIsmUqff"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "634cbce3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "634cbce3",
        "outputId": "fb1c2cf7-4c6d-4fc6-c34f-b703128c5171"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Spark NLP Version : 3.4.1\n",
            "Spark NLP_JSL Version : 3.4.1\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "import os\n",
        "from pyspark.ml import Pipeline,PipelineModel\n",
        "from pyspark.sql import SparkSession\n",
        "\n",
        "from sparknlp.annotator import *\n",
        "from sparknlp_jsl.annotator import *\n",
        "from sparknlp.base import *\n",
        "import sparknlp_jsl\n",
        "import sparknlp\n",
        "import pyspark.sql.functions as F\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "params = {\"spark.driver.memory\":\"6G\", \n",
        "          \"spark.kryoserializer.buffer.max\":\"2000M\", \n",
        "          \"spark.driver.maxResultSize\":\"2000M\"} \n",
        "\n",
        "spark = sparknlp_jsl.start(license_keys['SECRET'],params=params)\n",
        "\n",
        "print (\"Spark NLP Version :\", sparknlp.version())\n",
        "print (\"Spark NLP_JSL Version :\", sparknlp_jsl.version())"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "spark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 219
        },
        "id": "a6qhUXHwIm3i",
        "outputId": "47ec00bf-5f0f-4aac-ec41-992ac58d6938"
      },
      "id": "a6qhUXHwIm3i",
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://68edf19deb3f:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.1.2</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>Spark NLP Licensed</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ],
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7fc66d234090>"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Relationship extraction models are mostly very domain-specific and trained to detect only specific types of links. For this example, I have decided to include two John Snow Labs models in the NLP pipeline. One model will detect adverse drug effects between drugs and conditions, while the other model is used to extract relations between drugs and proteins."
      ],
      "metadata": {
        "id": "VPa31njyUvHY"
      },
      "id": "VPa31njyUvHY"
    },
    {
      "cell_type": "code",
      "source": [
        "documenter = DocumentAssembler()\\\n",
        "    .setInputCol(\"text\")\\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "sentencer = SentenceDetector()\\\n",
        "    .setInputCols([\"document\"])\\\n",
        "    .setOutputCol(\"sentences\")\n",
        "\n",
        "tokenizer = Tokenizer()\\\n",
        "    .setInputCols([\"sentences\"])\\\n",
        "    .setOutputCol(\"tokens\")\n",
        "\n",
        "# PoS and Dependency parser\n",
        "\n",
        "pos_tagger = PerceptronModel()\\\n",
        "    .pretrained(\"pos_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"sentences\", \"tokens\"])\\\n",
        "    .setOutputCol(\"pos_tags\")\n",
        "\n",
        "dependency_parser = DependencyParserModel()\\\n",
        "    .pretrained(\"dependency_conllu\", \"en\")\\\n",
        "    .setInputCols([\"sentences\", \"pos_tags\", \"tokens\"])\\\n",
        "    .setOutputCol(\"dependencies\")\n",
        "\n",
        "# NER for ReDL\n",
        "\n",
        "redl_words_embedder = WordEmbeddingsModel()\\\n",
        "    .pretrained(\"embeddings_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols([\"sentences\", \"tokens\"])\\\n",
        "    .setOutputCol(\"redl_embeddings\")\n",
        "\n",
        "redl_drugprot_ner_tagger = MedicalNerModel.pretrained(\"ner_drugprot_clinical\", \"en\", \"clinical/models\")\\\n",
        "    .setInputCols(\"sentences\", \"tokens\", \"redl_embeddings\")\\\n",
        "    .setOutputCol(\"redl_ner_tags\")\n",
        "\n",
        "redl_ner_converter = NerConverter()\\\n",
        "    .setInputCols([\"sentences\", \"tokens\", \"redl_ner_tags\"])\\\n",
        "    .setOutputCol(\"redl_ner_chunks\")\n",
        "\n",
        "# NER for ADE\n",
        "\n",
        "ade_words_embedder = BertEmbeddings() \\\n",
        "    .pretrained(\"biobert_pubmed_base_cased\", \"en\") \\\n",
        "    .setInputCols([\"sentences\", \"tokens\"]) \\\n",
        "    .setOutputCol(\"ade_embeddings\")\n",
        "\n",
        "ade_ner_tagger = MedicalNerModel() \\\n",
        "    .pretrained(\"ner_ade_biobert\", \"en\", \"clinical/models\") \\\n",
        "    .setInputCols([\"sentences\", \"tokens\", \"ade_embeddings\"]) \\\n",
        "    .setOutputCol(\"ade_ner_tags\")\n",
        "\n",
        "ade_ner_converter = NerConverter() \\\n",
        "    .setInputCols([\"sentences\", \"tokens\", \"ade_ner_tags\"]) \\\n",
        "    .setOutputCol(\"ade_ner_chunks\")\n",
        "\n",
        "# ReDL relaton extraction\n",
        "\n",
        "# Set a filter on pairs of named entities which will be treated as relation candidates\n",
        "drugprot_re_ner_chunk_filter = RENerChunksFilter()\\\n",
        "    .setInputCols([\"redl_ner_chunks\", \"dependencies\"])\\\n",
        "    .setOutputCol(\"redl_re_ner_chunks\")\\\n",
        "    .setMaxSyntacticDistance(4)\n",
        "    \n",
        "drugprot_re_Model = RelationExtractionDLModel()\\\n",
        "    .pretrained('redl_drugprot_biobert', \"en\", \"clinical/models\")\\\n",
        "    .setPredictionThreshold(0.9)\\\n",
        "    .setInputCols([\"redl_re_ner_chunks\", \"sentences\"])\\\n",
        "    .setOutputCol(\"redl_relations\")\n",
        "\n",
        "# ADE relation extraction\n",
        "\n",
        "ade_re_model = RelationExtractionModel()\\\n",
        "        .pretrained(\"re_ade_biobert\", \"en\", 'clinical/models')\\\n",
        "        .setInputCols([\"ade_embeddings\", \"pos_tags\", \"ade_ner_chunks\", \"dependencies\"])\\\n",
        "        .setOutputCol(\"ade_relations\")\\\n",
        "        .setMaxSyntacticDistance(3)\\\n",
        "        .setPredictionThreshold(0.9)\\\n",
        "        .setRelationPairs([\"drug-ade\"]) # Possible relation pairs. Default: All Relations.\n",
        "\n",
        "# Define whole pipeline\n",
        "pipeline = Pipeline(\n",
        "    stages=[documenter, sentencer, tokenizer,\n",
        "            pos_tagger,\n",
        "            dependency_parser,\n",
        "            redl_words_embedder,\n",
        "            redl_drugprot_ner_tagger,\n",
        "            redl_ner_converter,\n",
        "            ade_words_embedder,\n",
        "            ade_ner_tagger,\n",
        "            ade_ner_converter,\n",
        "            drugprot_re_ner_chunk_filter,\n",
        "            drugprot_re_Model,\n",
        "            ade_re_model])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7FAwuERKMM0t",
        "outputId": "c0a9e89a-52f0-436e-a9f2-dca425ef963b"
      },
      "id": "7FAwuERKMM0t",
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pos_clinical download started this may take some time.\n",
            "Approximate size to download 1.5 MB\n",
            "[OK!]\n",
            "dependency_conllu download started this may take some time.\n",
            "Approximate size to download 16.7 MB\n",
            "[OK!]\n",
            "embeddings_clinical download started this may take some time.\n",
            "Approximate size to download 1.6 GB\n",
            "[OK!]\n",
            "ner_drugprot_clinical download started this may take some time.\n",
            "Approximate size to download 14 MB\n",
            "[OK!]\n",
            "biobert_pubmed_base_cased download started this may take some time.\n",
            "Approximate size to download 386.4 MB\n",
            "[OK!]\n",
            "ner_ade_biobert download started this may take some time.\n",
            "Approximate size to download 15.3 MB\n",
            "[OK!]\n",
            "redl_drugprot_biobert download started this may take some time.\n",
            "Approximate size to download 386.6 MB\n",
            "[OK!]\n",
            "re_ade_biobert download started this may take some time.\n",
            "Approximate size to download 17.1 MB\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Some of the steps are relevant for both the ADE (Adverse Drug Effect) and REDL (Drugs and Proteins) relations. However, since the models detect relationships between different types of entities, we have to use two NER models to detect both types of entities. Then we can simply feed those entities into relation extraction models. For example, the ADE model will produce only two types of relationships (0,1), where 1 indicates an adverse drug effect. On the other hand, the REDL model is trained to detect nine different types of relations between drugs and proteins (ACTIVATOR, INHIBITOR, AGONIST…)."
      ],
      "metadata": {
        "id": "5rSERicNU6EL"
      },
      "id": "5rSERicNU6EL"
    },
    {
      "cell_type": "code",
      "source": [
        "def extract_rel_params(df):\n",
        "  \"\"\"\n",
        "  Extract relationship parameters from the output dataframe for ADE and ReDL relations\n",
        "  \"\"\"\n",
        "  rel_params = list()\n",
        "  for i, row in df.iterrows():\n",
        "      node_id = row['nodeId']\n",
        "      if row['redl_relations']:\n",
        "          for result in row['redl_relations']:\n",
        "              rel_type = result['result'].replace('-', '_')\n",
        "              confidence = result['metadata']['confidence']\n",
        "              entity_1_type = result['metadata']['entity1']\n",
        "              entity_1_label = result['metadata']['chunk1']\n",
        "              entity_2_type = result['metadata']['entity2']\n",
        "              entity_2_label = result['metadata']['chunk2']\n",
        "\n",
        "              rel_params.append({'node_id': node_id, 'rel_type': rel_type, 'confidence': confidence,\n",
        "                                'entity_1_type': entity_1_type, 'entity_1_label': entity_1_label, 'entity_2_type': entity_2_type, 'entity_2_label': entity_2_label})\n",
        "      if row['ade_relations']:\n",
        "          for result in row['ade_relations']:\n",
        "              # Skip when ADE is not found\n",
        "              if result['result'] == '0':\n",
        "                  continue\n",
        "              rel_type = 'ADE'\n",
        "              confidence = result['metadata']['confidence']\n",
        "              entity_1_type = result['metadata']['entity1']\n",
        "              entity_1_label = result['metadata']['chunk1']\n",
        "              entity_2_type = result['metadata']['entity2']\n",
        "              entity_2_label = result['metadata']['chunk2']\n",
        "\n",
        "              rel_params.append({'node_id': node_id, 'rel_type': rel_type, 'confidence': confidence,\n",
        "                                'entity_1_type': entity_1_type, 'entity_1_label': entity_1_label, 'entity_2_type': entity_2_type, 'entity_2_label': entity_2_label})\n",
        "\n",
        "  return rel_params\n"
      ],
      "metadata": {
        "id": "DQNNUhEbzsm7"
      },
      "id": "DQNNUhEbzsm7",
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lastly, we need to define the graph model to represent extracted entities. Mostly, it depends if you want the extracted relationships to point to their original text or not."
      ],
      "metadata": {
        "id": "qbF5PAuBU8yv"
      },
      "id": "qbF5PAuBU8yv"
    },
    {
      "cell_type": "code",
      "source": [
        "# Define neo4j import query\n",
        "import_rels_query = \"\"\"\n",
        "UNWIND $data AS row\n",
        "MATCH (a:Section)\n",
        "WHERE id(a) = toInteger(row.node_id)\n",
        "WITH row, a \n",
        "CALL apoc.merge.node(\n",
        "  ['Entity', row.entity_1_type],\n",
        "  {name: row.entity_1_label},\n",
        "  {},\n",
        "  {}\n",
        ") YIELD node AS startNode\n",
        "CALL apoc.merge.node(\n",
        "  ['Entity', row.entity_2_type],\n",
        "  {name: row.entity_2_label},\n",
        "  {},\n",
        "  {}\n",
        ") YIELD node AS endNode\n",
        "\n",
        "MERGE (startNode)-[:RELATIONSHIP]->(rel:Relationship {type: row.rel_type})-[:RELATIONSHIP]->(endNode)\n",
        "\n",
        "MERGE (a)-[:MENTIONS]->(startNode)\n",
        "MERGE (a)-[:MENTIONS]->(endNode)\n",
        "MERGE (a)-[rm:MENTIONS]->(rel)\n",
        "SET rm.confidence = row.confidence\n",
        "\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "sKJ7Xgg11SKs"
      },
      "id": "sKJ7Xgg11SKs",
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The only thing left is to execute the code and import extracted biomedical relations into Neo4j."
      ],
      "metadata": {
        "id": "5tUVJn5gU1xY"
      },
      "id": "5tUVJn5gU1xY"
    },
    {
      "cell_type": "code",
      "source": [
        "from datetime import datetime\n",
        "\n",
        "# Define NLP input\n",
        "nlp_input = run_query(\"\"\"\n",
        "MATCH (t:Section)\n",
        "RETURN id(t) AS nodeId, t.text as text\n",
        "LIMIT 1000\n",
        "\"\"\")\n",
        "\n",
        "# Run through NLP pipeline and store results\n",
        "step = 100  #batch size\n",
        "for i in range(0, len(nlp_input), step):\n",
        "  print(f\"Start processing row {i} at {datetime.now()}\")\n",
        "  # Create a chunk from the original Pandas Dataframe\n",
        "  chunk_df = nlp_input[i: i + step]\n",
        "  # Convert Pandas into Spark Dataframe\n",
        "  sparkDF=spark.createDataFrame(chunk_df)\n",
        "  # Run through NLP pipeline\n",
        "  result = pipeline.fit(sparkDF).transform(sparkDF)\n",
        "  df = result.toPandas()\n",
        "  # Extract REL params\n",
        "  rel_params = extract_rel_params(df)\n",
        "  # Store to Neo4j\n",
        "  run_query(import_rels_query, {'data': rel_params})\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ri0XZ3U_EDNa",
        "outputId": "86c285c4-1448-426e-d17d-0d15cfd774b7"
      },
      "id": "ri0XZ3U_EDNa",
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start processing row 0 at 2022-03-10 15:13:46.156199\n",
            "Start processing row 100 at 2022-03-10 15:23:28.880193\n",
            "Start processing row 200 at 2022-03-10 15:32:36.873671\n",
            "Start processing row 300 at 2022-03-10 15:42:27.586173\n",
            "Start processing row 400 at 2022-03-10 15:49:43.298054\n",
            "Start processing row 500 at 2022-03-10 15:55:56.895929\n",
            "Start processing row 600 at 2022-03-10 16:02:53.612173\n",
            "Start processing row 700 at 2022-03-10 16:09:38.976272\n",
            "Start processing row 800 at 2022-03-10 16:13:57.521061\n",
            "Start processing row 900 at 2022-03-10 16:18:28.480545\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This code processes only 1000 sections, but you can increase the limit if you want. Since we didn't specify any unique id of the Section nodes, I've fetched the text and section internal node ids from Neo4j, which will make the import of relations faster as matching nodes by long text is not the most optimized way. Usually, you can get around this problem by calculating and storing a hash of text like sha1. In Google Colab, it takes about an hour to process 1000 sections.\n",
        "\n",
        "Now we can examine the results. First, we will look at the relationships with the most mentions."
      ],
      "metadata": {
        "id": "vZ8K8ta9VAV7"
      },
      "id": "vZ8K8ta9VAV7"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (start:Entity)-[:RELATIONSHIP]->(r)-[:RELATIONSHIP]->(end:Entity)\n",
        "WITH start, end, r,\n",
        "  size((r)<-[:MENTIONS]-()) AS totalMentions\n",
        "ORDER BY totalMentions DESC\n",
        "LIMIT 5\n",
        "RETURN start.name AS startNode, r.type AS rel_type, end.name AS endNode, totalMentions\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "7Nbdl3fBsVEb",
        "outputId": "913c2104-2723-438f-a5ec-d2a0976abda4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "id": "7Nbdl3fBsVEb",
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-dc00be01-7240-4142-b13d-090995818853\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>startNode</th>\n",
              "      <th>rel_type</th>\n",
              "      <th>endNode</th>\n",
              "      <th>totalMentions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>cytokines</td>\n",
              "      <td>INDIRECT_UPREGULATOR</td>\n",
              "      <td>chemokines</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>cytokines</td>\n",
              "      <td>INDIRECT_UPREGULATOR</td>\n",
              "      <td>tumor necrosis factor-alpha</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>nitric oxide</td>\n",
              "      <td>PRODUCT_OF</td>\n",
              "      <td>NO</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>IL-1b</td>\n",
              "      <td>INDIRECT_UPREGULATOR</td>\n",
              "      <td>IL-6</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>matrix metalloproteinases</td>\n",
              "      <td>ACTIVATOR</td>\n",
              "      <td>MMPs</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dc00be01-7240-4142-b13d-090995818853')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-dc00be01-7240-4142-b13d-090995818853 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-dc00be01-7240-4142-b13d-090995818853');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                   startNode              rel_type  \\\n",
              "0                  cytokines  INDIRECT_UPREGULATOR   \n",
              "1                  cytokines  INDIRECT_UPREGULATOR   \n",
              "2               nitric oxide            PRODUCT_OF   \n",
              "3                      IL-1b  INDIRECT_UPREGULATOR   \n",
              "4  matrix metalloproteinases             ACTIVATOR   \n",
              "\n",
              "                       endNode  totalMentions  \n",
              "0                   chemokines              4  \n",
              "1  tumor necrosis factor-alpha              3  \n",
              "2                           NO              3  \n",
              "3                         IL-6              3  \n",
              "4                         MMPs              2  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since I am not a medical doctor, I won't comment the results as I have no idea how accurate they are. If we were to ask a medical doctor if a specific relation is valid, we can present them with the original text and let them decide."
      ],
      "metadata": {
        "id": "ScmGI6aaVD7g"
      },
      "id": "ScmGI6aaVD7g"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (start:Entity)-[:RELATIONSHIP]->(r)-[:RELATIONSHIP]->(end:Entity)\n",
        "WHERE start.name = 'cytokines' AND end.name = 'chemokines'\n",
        "MATCH (r)<-[:MENTIONS]-(section)<-[:HAS_SECTION]-(article)\n",
        "RETURN section.text AS text, article.pmid AS pmid\n",
        "LIMIT 5\n",
        "\"\"\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "DYMttKv--GOm",
        "outputId": "78a9c347-7ad3-4084-8abc-3a9b30e11b8d"
      },
      "id": "DYMttKv--GOm",
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-c9e00d6e-1550-4b81-9292-ffba989b1e7b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>pmid</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Chronic Lymphocytic Leukemia (CLL) is a B cell...</td>\n",
              "      <td>22202043</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>The developing brain is susceptible to hypoxic...</td>\n",
              "      <td>21622239</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>The two major neuropathologic hallmarks of AD ...</td>\n",
              "      <td>21196374</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Macrophages are versatile cells involved in he...</td>\n",
              "      <td>19273336</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c9e00d6e-1550-4b81-9292-ffba989b1e7b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c9e00d6e-1550-4b81-9292-ffba989b1e7b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c9e00d6e-1550-4b81-9292-ffba989b1e7b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                                text      pmid\n",
              "0  Chronic Lymphocytic Leukemia (CLL) is a B cell...  22202043\n",
              "1  The developing brain is susceptible to hypoxic...  21622239\n",
              "2  The two major neuropathologic hallmarks of AD ...  21196374\n",
              "3  Macrophages are versatile cells involved in he...  19273336"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "What might also be interesting is to search for indirect relationships between specific entities."
      ],
      "metadata": {
        "id": "PFjiw_qMVHlt"
      },
      "id": "PFjiw_qMVHlt"
    },
    {
      "cell_type": "code",
      "source": [
        "run_query(\"\"\"\n",
        "MATCH (start:Entity), (end:Entity)\n",
        "WHERE start.name = \"cytokines\" AND end.name = \"CD40L\"\n",
        "MATCH p=allShortestPaths((start)-[:RELATIONSHIP*..5]->(end))\n",
        "RETURN [n in nodes(p) | coalesce(n.name, n.type)] AS result LIMIT 25\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "P1UCcMOkvUxF",
        "outputId": "034eaf37-c140-4794-f109-877e54d87755",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        }
      },
      "id": "P1UCcMOkvUxF",
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-b4554600-5496-4171-b704-17bb1670aa33\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>[cytokines, INDIRECT_UPREGULATOR, IL-1b, INDIR...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b4554600-5496-4171-b704-17bb1670aa33')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b4554600-5496-4171-b704-17bb1670aa33 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b4554600-5496-4171-b704-17bb1670aa33');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                              result\n",
              "0  [cytokines, INDIRECT_UPREGULATOR, IL-1b, INDIR..."
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Next steps\n",
        "There are a couple of options we have to enhance our NLP pipeline. The first that comes to mind is using entity linking or resolver models. Basically the entity resolver maps an entity to a target knowledge base like UMLS or Ensembl. By accurately linking entities to a target knowledge base we achieve two things:\n",
        "* Entity disambiguation\n",
        "* Ability to enrich our knowledge graph with external sources \n",
        "\n",
        "For example, I've found two nodes entities in our graph that might refer to the same real-world entity. While John Snow Labs offers multiple Entity Resolution models, it takes a bit of domain knowledge to map entities to a specified target knowledge base efficiently. I've seen some real-world biomedical knowledge graphs that use multiple target knowledge bases like UMLS, OMIM, Entrez to cover all types of entities.\n",
        "The second feature of using entity resolvers is that we can enrich our knowledge graph by using external biomedical sources. For example, one application would be to use a knowledge base to import existing knowledge and then find new relations between entities through NLP extraction.\n",
        "Lastly, you could also use various graph machine learning libraries like the Neo4j GDS, PyKEEN, or even PyTorch Geometric to predict new relationships.\n",
        "\n",
        "Let me know if you find any exciting application using the combination of NLP pipelines and graph databases. Also let me know if you have some suggestions to improve any of the NLP or Knowledge Graph steps in this post. Thanks for reading!"
      ],
      "metadata": {
        "id": "_pnIWIrCVQyD"
      },
      "id": "_pnIWIrCVQyD"
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "1UNqPJP8VKfI"
      },
      "id": "1UNqPJP8VKfI",
      "execution_count": 20,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "colab": {
      "name": "Pubmed NLP.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}